Physical activity has been shown to be inversely associated with breast cancer recurrence and survival. Although physical activity is known to decline with age, rates of change in physical activity have not been well characterized in breast cancer patients and subgroups with known disparities in breast cancer survival, especially in minorities, the elderly, and the obese. We evaluated moderate and strenuous physical activity from high school through diagnosis in 1,220 breast cancer patients, and from high school to recruitment in 935 controls. We compared the proportion of patients and controls meeting the American Cancer Society (ACS) guidelines for physical activity and differences in declines in level of physical activity by race, age, and obesity. At diagnosis, only 33.2% of breast cancer patients met the ACS physical activity guidelines. Only 13.2, 24.7, and 30.5% of African-American (AA), obese, and older (≥65 years) patients met the guidelines, respectively. Controls showed slightly higher rates, with 36.4% overall, 23.7% of AA, 29.0% of obese, and 32.4% of older women meeting the guidelines. AA patients were less likely to meet guidelines compared to White patients (p < 0.0001). Obese patients were less likely to meet guidelines compared to non-obese (p < 0.0001). We found that both moderate and strenuous physical activity declined after high school in patients and controls. AA patients reported steeper declines in strenuous (p = 0.0027), and total (p = 0.0009) physical activity compared to Whites. Obese patients reported steeper declines in total physical activity compared to non-obese (p = 0.022). Differences in average slopes of declines in physical activity were not observed by age. Our results suggest that strategies and programs to encourage women to maintain recommended levels of physical activity after high school are needed. Furthermore, breast cancer patients, particularly AA and obese patients, should be targeted to help reduce disparities.